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ABSTRACT 

Methodology for input evaluation, as defined by 
Daniel L. Stuff lebeam, is relatively nonexistent. Advocate teams have 
recently become a popular means of generating and assessing 
alternative strategies for a set of objectives. This study was 
undertaken to develop and evaluate methodology for advocate team use 
m input evaluation. Steps taken included conducting a series of four 
case studies where advocate teams were used, development of a 
conceptualization and exemplary operationized procedures for advocate 
team use, and an evaluation of the proposed methodology through 
written critiques and empirical application. The methoc3ology wa-? 
revised^ based on the evaluation results, (Author) ' 
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Ii'ITP.ODUCtlON 

■ Major braakthroughs have bean made in educatidnal evaiua- 
tion theory develcpmant during the past ten years. However, 
a great need still exists fo^ evaluation methodology. This 
paper deEcrlbes a studj; in which mcthodglcgy for Input evalua- ■ 
tion uoing advocate teams was CLaated and evaluated. 

Input evaluation is the "I" in the CIPP evaluation 
framework developed by Daniel L. Stuff lebeara. Specifically, 
input evaluation provides 

. . .inforiiiation for detartnrt.ning how to utilise 
.cesources to rafjot program goals. This is accom- 
plishad by idrantifying and afloesalng 1) relevant 
capabilities of tha cesponBlble syateni, ■ 
2} atfategies for achieving progtara goals , and 
3} designs for Implementing a selected strategy. 1 

The value of input evaluation in education- is evident. 

Often in the past a person or porhaps a group of people have 

designed a program based solely on their past eKperience and 

common senc« . There is no systematic look at the resoutoes 

of the agency, nor is any effort made to identify or develop 

alternative pfogranis, let alone assess the alternative 

prograras in terms of pre-determlned criteria. 

The input evaluation framework can be usod to, identify 

and aaseoa fcncwn available alternatives. However, sometimas 

known alternatives do not exiat and therefore must be created. 



1 

^ . ^ Daniel L. Stuff lebeam, et_ a^. , Educ^=itienal Svaluation 
B^giffi Makina. (Ita«ca, IlllnolarTr-X^^aaJoci;--— 
Publishats, Tnc, 1971), p. 222. 
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When this happana, advocate teams provide a vital function in 
the input evaluation framework, in that they creatg_ alternative 
neans for accomplishing a given set of objectives. 

Advocate teams generally refer to groups of people 
specifically Selected and oriented to develop alternative 
strategies. The terra "advocate," however, needs further 
ej^lanation, A eriterion for selection of team members may 
that the members • hold a certain philosophy or "advocate" 
a partlGular position before they engage in strategy development. 
For example, in the development of an elementary school program 
for migrant students, one team may consist of ej^erts who 
advocate behavior modification, while a oonpeting team piay 
be composed of "British Infant school" advocates. The 
advocate team idea and the input evaluation framewack provided 
the boundaries within which methodQlogy was created. 

The study described In this paper contained three major 
parts. First, a "state of the art" picture of advocate team 
use was ebtAined by conducting four case studies on previous 
uses of advocate teams. The case results were analyzed and 
relevant literature was surveyed. Second, a technical manual 
was developed which included a basic conceptualization as ' 
well as exemplary operatlonaliaed procedures for doing input 
evaluation using advocate teams. Ihird, the nanuaJ. was 
evaluated In terms of its conceptual adequaey and practical 
utility. Ea<m of these three activities— the case studies. 



iJavelopment of the technical manual, and evaluation of the 
manual-will be described in the following sections . 

CASS STUDISS 

rhe case study approach is extwraely valuable when relativel 
little is known about the area-^in this case, bow advocate teams 
really work or how one would actually go about doing input 
evaluation. The four cases .elected for study satisfied the 
criteria of relevance and feasibility, m addition, the 
four case studies were, at the beginning of this study , ^^he 
only major efforts to use advocate tean« that have been identic 
fied. The four cases were: 

1. Alternative Strategy Development for Secondary 
^^^t Students, conducted by the Southwest 
Educational Development Laboratory, 1969, 

^' tl^e Development of The Ohio 

State University Model Training Progtara. 

Cen?^r!l9^. ^" University Valuation 

^" th^n? "*n"^nfl and Assessment System for 

the Division of tonpow«r and Institutions, 

4. A Repianning Study of the Illinois Plan for Proaram 
T^rin?""?* Gi't^d Children, conducted by 
^^i^!''''''"''' EducatiLal 

the procedures used in conducting the case studies will be 
presented, followed by a summary of the analysis, " 
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Precaduees 

In order to set boundasles foe the case analysis and 
collection of data, the foUowing dlraMsions wete specified 
prior to conducting the case studies. 

1. Substantive content - The ease studies pcoyided 

a tracing o£ ideas and events relatlno to advocate 
team use. xha content included an intensive 
look into Identification of the methodology used, 
how xt was developed, and the participantr 
involved # 

w^f user*'?;\" ^ """^"^^ retrospective time line 
ntl Jhf 'h the decision was made to 

use the advocate team approach and ended when the 
strategies developed were at the point of being 

BoSi— * / beginning of the study 

sumnacwed relevant antecedent events. A brief 

su«m^°" *M ef the study, labeled "Aftermath, 

summarised relevant events during implenentationr 

3. Richness of detail - Within the case studies. 

richness of detail was less for historical events 
which were not directly related to developwenfof 
the methodology itself and how it wa. used? L 
like ^manner, processes not directly related to 

ti'TJ"" °- methodology and it. use 
were treated more superficially, 

^' ^^^^6!i^'"''"''^'i^'r Within this dimension. 
• two differences within case atudles. For 

an the investigator played the .ole of 

an Butside observer, coning on the case after it 
was conpleted. For the r«maini„g two cas« 4udL. 
if Plftyed a part during tha usage ' 

^* 1oflhe*c^rn*S? - »tKl« a£ reporting seie^oid 
n^L!?! studies included narrations, summary 

presentations, and eKhlblts. unmary 

6. - Culbertson, Jacobson. and Reller have 

Usfd and desoribed five perspective, that can^ 
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used, either single off In combination, in 
developing a cassy^ a combination of fEamewoL-ks 
was selected as most appEopciate for this case 
study methodolofly. The fraraewoirk was historical , 
in that case events waire described In a chconologi- 
cal series. The case studies were thematic , in that 
the case was built around the particular theme of 
advocate tmm methodology and use. Finally, the 
case approach included the p roc ess framework, is it 
focuaed upon group and individual interactions in 
the development and use of advocate team methodology. 

In addition to detecroining dimenslona of the case studies, 
a systematic sequenGe of activities was outlined, A work 
breakdown schedule and a checklist of activities were developed 
prior to the initiation of the case studies and were used as 
guides for each ease study. 

Various pror.edures were used to increase the eaie study 
. validity and eeliahilityj e.g., a variety of sources of data 
were used| primary and original sourees were used when possible, 
as opposed to second-hand data sources | a system for Identifica- 
tion and solution of data dlserepancles vmm developed and used. 

The case studies were derived following the f ranawork 
3'uat described. Over two hundred fifty documents were analysed, 
and extensive interviews were conducted with approximately 
thirty persons,. 

Case Study Analysis 

Looking across case studies in terms of general character- 
istics, several things ware obseryable. First, two uses of 



2 

Adml 
Pr 



2jack Culbertaon, Paul Jacobson, ancS Theodore Seller, 
ministratlvj. gelationship;; (Englewood Cliffs, Hew Jmi:mmvi 
entice-HaU, 1960), pp. 63-64. 
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advocat. teams sponaored at the state level, t*.o at the 

federal level. Second, all adyooate teams were used to create 
strategies for an area in which acceptable alternatives were 
not available. Third, the programs ereated by advocate teams 
in these ease studies were to bt Implemented in a variety of 
settings I a research. and development laboratory, .university, 
a state department, and a federal agency. 

When roles across the case studies were analysed, several 
things were noticeable. 

1. All cases had a group or team of people directly 
alvoer^'^i^ Electing, orienting, and running 

^' ^^t-^'^^l:'^^ ^y***"" decision makers in 

some fashion? however, degree of Involvement varied. 

3. In all cases a group of people assessed the 
b™^^''*^ through sorae procedure, although the 
^aSlS^w'^"^ ^"if fl""!^- in all cises assessment 
Altuii 4^"^'?*^ *° •J''**" decision makers. 
Actual decisions about aelection of strategies or 
convergence of strat.giea war. made by thele delLion 

4. All cases used advocate teams. One case used 
advocate tea^ twice for two different purposes , 
Scausf ""'^^^^ selected team members 
program development. Knowledfle of these fact. " 
aefined in the technical manaal. 

Since the pu^o„ of this study to design a technical 
manual for input evaluation using advocate and design t.^ms, 
analysos of the ease studies i„ terms of how acttvitie. related 
to input evaluation functiom\,.. «rli.ia«i. flrait, v.a« 



variance in the degree to whl<ai advocatg teams followed an 
input evaluation framework. One case study contained almost all 
of the functions within input evaluation, whereas two others 
containsd only a few elements (identification of capabilities 
and identification of alternative strategies). Second, most 
of the infoirmation obtained on advocate team use in all cases 
centered around the identlf ication responsibilities of ' 
advocate teams. Although some information on assessing 
strategies was obtained, little information on system capabill- 
tt4U identified. Also, the design function of input evalua- 
tion was present in only one ease study. 

Inforaation obtained about tht: identification aspect of 
advocate team use can be further delineated Into a series of 
generalizations across case studies, which were taken into 
consideration in development of the technical manual. 'I'hese 
generalizations were as followsi . 

1. Selection of advocate team members was viewed as 
an extremely inportant process because It directly 
affects the output from the teams, 

2. Use of docuraents presented problems —gene rally, 
too many were used and often the documents were not 
viewed as relevant by team members, 

3. An orientation session was viewed as a useful way 

to orient team nenters . However .orientation sessions 
should provide^^ information to advocate team 
menbers and allow time for questions and answers, 

4. Advocate team sessions ranging from three to five 
days were cited as appr^riate, , 



5. Writing, sessions should be located where intassuptions 
ace not possible. Some, but not all, persons 
interviewed preferred a motel location to an office 
location. 

6. Assistance of administrative p«rsonnel during actual 
writing sessions will Increase the effectiveness 

of the team. 

t. Developers of specifications for advocate team use 
were eoncerned about the degree of specification— 
they wanted creative responses and yet they were con- 
cerned that the strategies would not be useful if no 
structure was provided, 

..... ifc.^ 

a. All advocate teaias felt free to deviate from the 
specifications if they could establish a ratiomae 
for the deviati.>ti» and often they did, 

9. The number of team members cited as appropriate 
ranged from three to six. 

In addition to generalizations across case studies, strength 
and weaknesses within each case study provided additional infor- 
mation for manual development. The strengths of the individual 
case studies were often included in the manual . For eKample, 
one of the strength, cited was the development of dissemination 
materials through which many reference groups could provide ■ 
evaluative it^ut about the three plans developed by advocate 
teams. Descriptions of these activities were presented in the 
manual as a possible assessment procedure. 

The weaknesses cited within cases were also used, in that 
processes were ineluded which hopefully would prevent the 
occurrence of the weakness in other use. of advocate teama. 
For example, in several case studies, a i« eg,, amount of 



information pectlnent to the tas;: assigned ta advocate teams 
was not used. In order to provide this needed infornkntion to 
advocats teams witnout placing untfealistic reading expectations 
on team membees, the cola of input evaluation resource person 
was created. This person would wor:i with advocate teams and 
retrisve and suraraariEe needed i.ifoemation upon request , a 
larce number of ideas, as well as actual content for the manual, 

was derived, from the case study analysis of strengths and weak- 

nesses. 

Finally, the ease studies were anaiyMd for another purpose, 
i.e., idantificLtio.i of areas wiare additional literature was 
neaded. ■ 

The technical manuAl was developed after completion of 
the analysis of case studies and the review of literature » The 
three liiajor activities involved in tha.tnftnual development were 
as follows t salsctio.v p£ a logical framsworl^i identification 
of key elements ,to be. included in the aanuaT^i I and writing 
and rewrtttnff. a^el, of these is d^^^ 

A nurabar of posoibie ftarieworks were surveyed for pos 
use in dewelopmertt of ,the manuaU These i.nGludad; the frame- 
woe" «fiiieW aiba afld Stuff labea.a used to describe an evaluation - 
unit (capability^ P|ocf#P£l *3^fteyj and ftredilbillty aofl 
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accepte.nce) ,2 Anotaer possible framework was that pcovided 
by Gerald Wadler in his systems work.'* Both of these ftamewocJis 
were rejectedj as the investigator felt that the framewocl: 
elements would need to be axplicated and use of these fraraewoclss 
might confuse the readers. Therefore, the investigator chose 
to use the rather simple framework of roles and functions. 
The roles and funetlons frameworl? seemed to fit into the 
content of the raanual without extensive ej^lanation of terras 
to the potential user. 

ICey elenieftts within the frameworU were derived from the 
case study analysis, related literature, and past es^erience of 
the investigator in use of advocate teams. Table 1 on page 11 
provides sunmary information about the sources of various manual 
eleinents^ ' 

The manual was revised four times after the original draft, 
based upon the critiques of students and staff at the OSU 
Evaluation Center. 

The manual was divided Into three major parts, each of 
which answered specific questions. The first part answered the 
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^gon asba and Daniel L. 3 tuff lebeamy "Stfategies foe the 
Institutionalization of th^ CIPP Evaluation MoiJaX" (An addrMs 
deliveced at the Bleventh Phi Delta itoppa Symposium on 
Bdueational Researeh| Columbus, Ohio, June 24, 1970), T. 

Strald Nadlar, Work Peeiqai A Syst#M Coficept (Madison, 
WiSG ois in I Richard D* Irwin, Inc^i 1966)^ 
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and What are the prior nff«^it,'„rt*, ^he use e£ input 

evaluation using advocate teanis i' 

The second part answered the queationi^ W^^ 
I need in order to use input evaluation employing advocate 
teams? Specif ie needs are discussed in terms of five roles. 

The third najor part ans the quastioni How do you do 

input •valuation? It Included a section on criteria, a flow- 
chart, and a series of overall guidelines artd pioeedural steps 
for each of the three functions of input evaluation. 

EVALUATION 

Evaluative data ori the techhical manual wer^^ 
» number of personi. Essentially, two major questions were 
answeeed through the ovaluat ion I 

1. What is the worth of the manual as it .now exists? 

2 i How can the manuai be improved? 

The sections which follow describe the evaluation ptoce- 
dutes used and present the «sults. Two separate groups provid^^ 
evaluative information . The first flro,^ consisted of the staff 

of a project which was currently using advocate teams. The 
second group (the critique panel ) consisted of peopio who had 
been involved in previous uses advocate teams. 
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Projceduges 

On Juljf 1, 1972, The Ohio State University Evaluation 
Center entered into a contract with the Ohio State Departraent 
of E^cation t® generate and assese alternative accountabllitj; 
models for the State of Ohio, Since this project included use 
o£ advocate teams, it seemed appropriate for the project staff 
to use the technical manual and provide evaluative feedback to 
the investigator on the worth of the manual and how it could 
be ii^roved. It should be noted that the project did not use 
the entire manual , In come cases luggest ions made in the manual 
were implemented by the staff j in other cases it was not 
possible to implement sections of the manual because of the 
uniqueness of this speeifie contract and because the staff did 
not have a copy of the manual until after the project was under 
way. 

Several means of data collection were used in the evaluation 
of the manual within the accountability pro|Ject, Pirst, tape 
recordings were made of the first day«s meeting of each advocate 
team. In addition to providing information on clarification 
needed by the teams, how they used their time, and irtiat document;, 
were referenced, the tapes provided a rich description of ways 
in which the three advocate teams worked. Second, the project 
staff was asked to keep a log of questions peitaining to the 
manual. Third, an extensive queationnaire was con^ietBd by 
accountability staff menbers. 
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A ctltlque panel consisting of twelve members was identified 
to critique the tecl-inlcal manuals Members of this panel were 
selected from persons who had had some involvement with advocate 
teams in the past. An exception was one person who had completed 
previous conceptual work on input evaluation, but was not 
familiar with advocate teams and their use. 

Members of this critique panel were categorized into four 
groups, each group having three members. The first group con- 
tained methodologists in general. Persons selected for this group 
had previously developed methodology for advocate team use or 
for input evaluation in Qeneral. The second group was composed 
of decision makers who had been involved in a previous use of 
advocate teams. The third group of persons were evaluate rs 
from previous uses of advocate teams. And the remaining group 
represented a selection of persons who had served on an advocate 
team in one of the previous uses. 

The questionnai re completed by the e ritique panel was 
divided into two parts. The first part contained a series of 
sixteen items about the manual or the methodology within the 
manual. The critiquers were Instructed to rate each item by 
placing an X In the box undetneath the description which most 
closely deseribed their attitude toward the manual or the 
methodology. A four-point scale ms used containing ratings 
ranging from "very poor" to "very good." A description of eaih 
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rating was present ad within the box to anchor the item# Th© 
second part of the questionnaire asked for Information on how 
the manual could be isprbved in general and/or in E'elation to 
specific seatlons* 

Results 

Several points are noteworthy in respanaea to the question^ 
"What is the worth of the manual?" h large number of high ratings 
were obtained from the accountability project staff (three persono 
on the six lections of the manual which they atteinpted to 
implement. Those sectiona were viewed aa very valuable to the 
staff. Results indicated that the staff considered the manual 
worthwhile and/that they attempted to implement it whenever 
possible^ They also viewed the manual ai very inst racial ve in 
using advocate teams. 

The questionnaire results provided by the critique panel 
supported the aecountabillty project staff 's perceptions of the 
manual 's value 6 Many more ratings of "good" or "very good" v^^:^:> 
ebtalned for the criteria used to evaluate the manual than ra^^.-r 
of "poor" or "very poor" (five to one ratio). The manual itmul: 
^nd its suggested methodology were seen as having utility to a 
potential user* Howevers examples used in the manual and the 
methodology atoptabillty need improvement. Overall findings 
indicated that there is a great need for a manual of this typs* 

"How can the manual be Improved?" Data obtained from 
responses to this question were of great importance to the 
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investigatoifi it is the intent of the investigator to proceed 
with the developtnental pcocess of the technical manual and to 
t-evlse it eKt«nsively6 Suggestions wece received such asi 
include more exAmplea> use a less formal writing styleiand 
provide suggestions on how elements of the input evaluation 
pfocess could be adapted to meet unique individual needs. 

corCLusicw " 

Ihe preceding paragraphs described a study in whieh 
methodology for Input evaluation using advocate teams was 
created and evaluated. The manual Is currently under revision 
by the inveotigator 0 A large number of areas surrounding 
input evaluation need additional investigation. Both conceptual 
and empieical work are needed. Howevep, input evaluation usino 
advocate teams has been performed successtfully and appears to 
be a valuable alternative for educational program development 
■ and assessment ,, 



